A model of colour appearance based on efficient coding of natural images

An object’s colour, brightness and pattern are all influenced by its surroundings, and a number of visual phenomena and “illusions” have been discovered that highlight these often dramatic effects. Explanations for these phenomena range from low-level neural mechanisms to high-level processes that incorporate contextual information or prior knowledge. Importantly, few of these phenomena can currently be accounted for in quantitative models of colour appearance. Here we ask to what extent colour appearance is predicted by a model based on the principle of coding efficiency. The model assumes that the image is encoded by noisy spatio-chromatic filters at one octave separations, which are either circularly symmetrical or oriented. Each spatial band’s lower threshold is set by the contrast sensitivity function, and the dynamic range of the band is a fixed multiple of this threshold, above which the response saturates. Filter outputs are then reweighted to give equal power in each channel for natural images. We demonstrate that the model fits human behavioural performance in psychophysics experiments, and also primate retinal ganglion responses. Next, we systematically test the model’s ability to qualitatively predict over 50 brightness and colour phenomena, with almost complete success. This implies that much of colour appearance is potentially attributable to simple mechanisms evolved for efficient coding of natural images, and is a well-founded basis for modelling the vision of humans and other animals.

The crispening effect causes perceived contrasts to be greater when the grey levels are nearer those of the background. The effect was modelled by Whittle [1], and subsequent work suggests the dipper effect [16] and divisive gain explains the effect [17]. Here we use Whittle's 1992 data to determine the dynamic range of human luminance vision.
Human subjects adjusted grey targets in equal-contrast steps on a grey background Generated from Whittle's Data [1] DoG fit, DR=15, R 2 =0. 994 Gabor fit, DR=3.75, R 2 =0.995 Both models outperform CIE L (R 2 =0.944) DoG fit when using the above DR R 2 = 0.946 Gabor fit when using the above DR, R 2 = 0.935 Both models outperform CIE L (R 2 =0.746) The ability of humans and other animals to perceive contrasts is dependent on the spatial frequency of those contrasts. Contrast sensitivity functions describe the contrast a of a sinwave that is detectable at different spatial frequencies. A related phenomenon is contrast constancy, where suprathreshold contrasts appear to be uniform irrespective of spatial frequency.

Contrast sensitivity functions
Sinewaves are generated with specific Michelson contrasts to ensure the model only permits detectable contrasts.
Removes subthreshold contrasts, matching CSF Suprathreshold sinewaves of different spatial frequencies should have equal amplitudes.
Suprathreshold contrast constancy is enhanced by saturation thresholds preventing multiplicative gain effects.
This family of illusions causes grey targets to differ in perceived brightness dependent on the arrangement of (typically high contrast) surrounds. Some of these illusions, such as simultaneous contrast and Mach bands have traditionally been attributed to centre-surround antagonism [18]. However the White illusions create the opposite effect, and have variously been attributed to oriented filtering with normalisation [3,19], T-junctions [e.g. 20], Gestalt/grouping/anchoring based mechanisms [5]. A further set of illusions have been attributed to 3D surface and lighting based inferences [see 20], or atmospheric-based inferences [see 20]. A grey bar flanked by black appears darker than the same grey flanked by white Adapted from [2] & [3] A grey square flanked by black squares appears darker than the same grey flanked by white squares Simultaneous brightness contrast The central grey bar is a uniform grey value, but the gradient in the background creates a powerful inverse luminance gradient in the bar. This is typically explained by centre-surround antagonism.
Both models create an inverse gradient, though the Gabor model's is more linear across the entire bar length.

Poggendorff illusion
Corrugated plaid Simultaneous brightness contrast A grey square surrounded by black appears darker than the same grey surrounded by white.
Adapted from [3] The steps in a sequence of grey levels from light to dark appear flat/homogeneous on a contrasting gradient, but when viewed against a matching gradient each step appears to have a strong internal gradient.
The internal gradients are much stronger in the lower rather than upper staircase The internal gradients are much stronger in the lower rather than upper staircase Chevreul staircase control Geier & Hudák (2011) find that the illusion persists when a counter-gradient surround is placed around the illusion, and suggest that traditional centre-surround antagonism cannot explain the effect.
Adapted from [4] As above, though the effect is not as powerful As above, though the effect is not as powerful Chevreul staircase control As above, however a white surround is found to eliminate the internal gradients of the staircases.
Still retains fairly clear internal gradients, although they are less powerful than above Still retains fairly clear internal gradients, although they are less powerful than above A light grid causes a grey rectangle to appear lighter than the same grey surrounded by a dark grid.
Illusory checkerboard patterns are created in a horizontal grey bar placed over a vertical grating.

Adapted from [3]
Circular variant of White's bar illusion. The grey ring neighbouring white rings appears lighter, and the same grey neighbouring dark rings appears darker.
Simultaneous brightness illusion that uses a background gradient.
Illusory stripes are created in a grey bar placed over a diagonal grating. Illusory stripes don't span the entire height of the bar The perceived brightness of identical grey patches on a checkerboard can be altered by various 3D and shading manipulations. The controls demonstrate how 3Dinference does not explain the effect [20].

Figures from [20]
Correctly   Dark, high contrast surrounds increase perceived brightness of the lower tile. Adelson attributes the effect to perceived atmospheric differences between the tiles. Adelson checker shadow illusion The shadow cast onto the checkerboard causes the shaded square to appear brighter than a square with the same grey level outside of the shadow.

Reverse contrast illusion
The grey diagonal bar surrounded by black bars and white background appears brighter, and the opposite is true for an inverted example.
The triangle cutting into the arm of the cross appears brighter than the triangle that spans between two arms.

Variant of White's bar illusion with zigzag background
Zaidi [7] provide a detailed study of the magnitude of various brightness induction effects. They use t-junction theory to model the predicted outcomes.
The study details the experimental viewing conditions, allowing for accurate representation and model testing.
Although the authors do quantify the magnitude of the effects (i.e. the brightness adjustments required to nullify the illusion) only two subjects were used, meaning it is impossible to quantify variance and therefore the magnitude of differences. Moreover, we note that the effects our Gabor model fails to predict well are also effects that are only marginally visible to us.       Mach bands [9] Contrast induction The study details the experimental viewing conditions, allowing for accurate representation and model testing.
Although the authors do quantify the magnitude of the effects (i.e. the brightness adjustments required to nullify the illusion) only two subjects were used, meaning it is impossible to quantify variance and therefore the magnitude of differences. Moreover, we note that the effects our Gabor model fails to predict well are also effects that are only marginally visible to us. Correctly predicts that dark spots should appear on the straightangled grid, but not with the wavy grid. The curved edges prevent the Gabor filters from bridging the gap between opposing corners.
A target's internal contrast is influenced by the contrast of its surrounds. The causes are unclear, though are generally thought to depend on local normalisation of contrasts.
Textural contrast induction Low contrast surrounds increase perceived target contrast, and this effect is most pronounced when the spatial frequency (SF) of the surround matches the target. In these example images the target on the left appears to have higher internal contrast than the same target on the right. The effect is most pronounced in the centre version with a matched spatial frequency.
Adapted from [10] Target contrast is enhanced on a low-contrast background, and most powerfully for SF-matched background. Target SD is enhanced 4%, 17% and 11% for high SF, matched SF, and low SF respectively.
Correctly predicts effect more powerfully than DoG (left). The target SD is enhanced 19%, 24% and 22% for high SF, matched SF, and low SF respectively.

[12]
Neutral image [13] Orientationdependent contrast induction ("tilt illusion") High contrast surrounds reduce perceived target contrast when texture orientations match. In the example here the upper target has bars aligned with the background (in phase). In the centre is the same target rotated 90 degrees (orientations mismatched), and it appears to have a higher contrast. We also include a final control where the aligned target is out of phase with the surround. This target also appears to have higher contrast than the in-phase upper target (implying the effect is not entirely controlled by orientation).
Custom figure with control, see [3] for similar effect.
Interestingly the DoG model (without orientation sensitivity) is able to simulate the effect, albeit weakly. Compared to the top, internal SD is 6% higher in the middle target, and 4% higher in the lower target.
The oriented model is able to predict the contrast induction effect. Compared to the top, internal SD is 10% higher in the middle target and 11% higher in the lower target.
Chromatic contrast induction High chromatic-contrast surrounds reduce perceived chromaticity. The high and lowcontrast surrounds have the same luminance, red-green, and blue-yellow background averages. The targets appear to be more colourful (higher chromaticity) in the lower image.
Adapted from [11] Chromaticity (average Euclidean distance from each target's colour to the background average) is 19% higher on the low contrast background.
Chromatic channels use DoG, so only luminance varies (same 19% chromatic induction effect as left). The model also predicts chromatic grating induction in the high contrast surround.

Colour constancy and chromatic adaptation
Colour constancy causes surfaces to appear to have the same colour under different lighting colours, generally attributed to chromatic adaptation. The mechanism by which this occurs is poorly understood, and models of whole scene averages, local surround averages and local maxima do not explain the effects fully [21].

Lotto, Purves & Nundy cube
The cube is rendered with different simulated lighting conditions; yellow-tinted and blue-tinted. Colour-constancy causes grey tiles to appear blue in the yellowtinted example, and yellow in the bluetinted example.
Models colour constancy effects (i.e. grey in the left becomes blue, grey on the right becomes yellow). Also models brightness induction effect.
Simulated chromatic adaptation of natural scene, here the linear red channel is multiplied by 5 Chromatic adaptation lets us (and other animals) estimate the colour of an object even as the colour of the illuminant shifts. So, for example, as illuminant colour alters with weather and time of day, objects appear to stay the same colour. The capacity for maintaining colour constancy through chromatic adaptation is limited at some point by saturation levels.

Generated example
Chromatic modelling only uses DoG, however in this case we use the Gabor model for luminance.
The model is largely robust against even comparatively extreme differences in a scene's simulated illumination colour. Nevertheless, the model will start to show differences when the colours become so extreme that they saturate some spatial frequencies more. e.g. here the lower image has more blue-yellow saturation. Another interesting feature of the model is that it does not result in scene normalisation -this green scene of a woodland is predicted to be green by the model (not average grey) Subtractive colour circles illusion This illusion places a cyan and magenta circle above a blue-white grating. The third circle is white, however simultaneous contrast makes it appear yellow. Spreading combines with the simultaneous contrast to make the intersection between cyan and white appear green.

Monnier & Shevell illusion
Colour assimilation is found to be more powerful (i.e. colour blending with its surrounds more powerfully) with a striped surround than with a solid surround. In this example the orange ring is identical in all five upper instances, however the spreading effect is more powerful for the ring surrounded by stripes, than the rings surrounded by the same solid colours.
Adapted from [14] Both models demonstrate powerful spreading effects, however they predict it should be more powerful with a solid surround.
When adjusting the model to give higher spatial frequencies a higher gain, this effect can be modelled correctly.
A number of the brightness illusions above are also powerful in a chromatic context (though not all). Interesting exceptions include illusory spots such as the Hermann grid (which our model suggests requires orientation-sensitive filters.

Chromatic Chevreul staircase
The concentric circles on the left appear to have internal gradients, but they are actually uniform flat colours. The black line surrounding the circles on the right eliminates the effect.
Adapted from [15] & [22] The model is able to simulate the gradients in the staircase, and the control does show flat steps (although the effect reduces toward the centre) The output figure here shows the RG signal, processed with a bandwidth of 5 Patterns increase perceived saturation Shapley et al. [22] show that a checker pattern (left) is perceived to have a higher saturation than the same colour averaged over a larger area (right), even though both have the same average cone stimulation.
We simulated Shapley et al.'s [22] data by multiplying the input image's RG signal by different values (graph's xaxis). The output RG signal for the checker pattern increases more than the area-averaged RG value (yaxis).